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Advantages  of  UAV 


•Lower  Cost  in  Manufacturing 
•Reduced  Cost  in  Maintenance  and  Operation 
•Energy  Saving  for  Smaller  Size 
•Minimal  Human  Risk 
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Aircraft  Landing  in  stormy  weather 
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Fly-by- Fee  I  Autonomous  Flight 


But  the  system  must  be: 
•Minimal  or  no  Weight  Increase 
•Low  Cost  in  Manufacturing 
•Robust  in  System  Integration 
•Easy  for  Installation 
•Friendly  in  Implementation 
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Bio-inspired  Smart  Materials/Structures 
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Meissner’s 

corpuscle 

To  measure  vibration 

Harr  Follicle 

To  measure:  flow, 
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Krause’s 

To  measure 
temperature 

Merkel’s  Disk: 

To  measure  toucf 


Massively 
Parallel  data 
processing, 
filtering, 
self-learning, 
diagnostics,  and 
real-time  decision 


MILLIONS  of 

nano/micro-sen 

sors, 

electronics, 
processing 
units  etc.  over  a 
large  area 


Pacinian  Corpuscle 

To  measure  touch 
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What  is  an  Intelligent  Material 
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Materials  Development 


Multifunctional  Materials 


Networks 


Processors/Neuron  circuits 


Multi-functional  Sensors 
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Sensor  Processing  Development 
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STAGE  2 

State  Quantification 


STAGE  1 

State  Classification 
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Signal  processing  +  machine 
learning  algorithms 
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Major  Tasks 


*  Bio-inspired  Sensor  Network 

-  Stretchable  sensor  network  to  accommodate  large  arrays  of  sensors 
and  electronics  over  a  large  area. 

*  Micro/Nano  Sensors  for  State  Sensing 

-  Multi-physic  multi-scale  sensors  with  an  ease  of  network  integration. 

*  Neuron  Circuits  and  Interface  Electronics 

-  Bio-inspired  neuron  circuits  with  appropriate  electronics  to  interface 
with  various  sensors. 

*  Modeling,  Design,  and  Prognostics 

-  Multi-physic  and  multi-scale  modeling  of  multifunctional  materials 
with  distributed  sensing  capabilities  for  design  and  validation. 

*  Diagnostics  and  State  Awareness 

-  Embedded  intelligent  software,  Algorithms,  tools,  and  processes  to 
determine  the  state  of  the  materials  in  real  time. 

*  Integration 

-  An  effort  to  develop  a  prototype  of  “intelligent  sensing  material.” 


AFOSR-MURI 

Bio-inspired  Sensory  Network 


NYlT 

New  York  Institute 
of  Technology 


UBC 


h..ft  jj.-Vy. 


Bio-inspired  Sensory  Network 


Chang,  Peumans/Wang  -  Stanford 
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Robust  and  Low  Cost  Materials  Developr 
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From  NANO  to  MACRO 


Step  5:  Training  and  Learning  step  4:  integration  and  Functionalization 


Making  network 


Step  3:  Network  Stretch  and  Expansion 


Adding  sensors/electronics 


CMOS/MEMS  Process 


CMOS  Process 

Step  1:  Stretchable  Network  Design 


Step  2:  Nano/Microsensors  and 
Electronics 


NANO-MICROSCALE  DESIGN  AND  FABRICATION  (CMOS  PROCESSES) 


by  G.  Lanzara 


OTFD  Sensors  for  Stretchable  Network 


Integration  of  Organic  Thin  Film  Diodes 
(OTFDS) 

-  Packaged  OTFDs  in  the  network 

•  Improved  diode  performance 

•  Protect  OTFDs  in  harsh  environment: 

-  High  temperature(350°C), 

-  Solvents,  acids 


-  To  measure  temperature 
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Temperature  (°  C) 

Temperature  measurement  (l~T)  of  an  OTFD 


An  OTFD  based  temperature  sensor  network 
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Video:  1 69  nodes  network  after  release 


Coating  3D  bodies 


Expanded  network 


•  t 


G.Lanzara,  J.  Feng  and  F.K.  Chang,  Smart  Materials  and  Structures,  19,  045013,  2010 


G.Lanzara  et  al,  Advanced  Materials,  2010 
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Fully  Stretched  Sensory  Network 
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before 

stretch 


Single  wire  resistance  unchanged  during/after  full  stretch  (400  Q) 
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Multi-physic  and  Multi-scale 

Sensors 


Chang,  Wang  -  Stanford 
McLeod  -  UC  Boulder 
Carman  -  UCLA 
Servati,  Ko  -  UBC 
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Network  Functionalization 


Airflow  sensors 


PZT  sensor  Carbon  nanotube 
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PZT  Actuators/Sensors  for  Stretchable  Network 


Create  piezoelectric  sensing 
systems  on  the  stretchable  network 

Method:  Integrate  thick  film 
ceramics  into  C-MOS  processing 

Challenges:  Temperature  & 
chemical  compatibility  issues 
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Recent  Accomplishments 


Screen  printed  piezo-ceramics  integrated  into  C-Mos  type  processing  & 
released  onto  a  polyimide  film  with  electrodes. 


Thick  film  piezos  on  a 


•  Innovations 


Piezos  released  onto  a  polyimide  film 


Piezos 


New  method  to  transfer  piezos  from  a  fabrication  substrate  to  an 
organic  substrate 

NYiT 
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Ongoing  Work 


•  Create  a  stretchable  network  from  the  screen  printed  piezos  released  onto 
an  organic  backing 


•  Characterization  of  materials 

•  New  transducer  designs 


Piezos  on  a 
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Air  Flow  Sensor  Configuration 


(Yue  Guo,  Prof.  Shan  X.  Wang) 

Aim:  Obtain  the  real-time  air  flow  profile  (velocity  +  direction)  surrounding  the  entire  airplane 


Volume:  Strain  tensor,  x  component  (1) 


1.  Air  Flow  hits  the  pillar 

2.  Deflection  in  the  beam 

3.  Strain  in  the  sensing  elements 

4.  Inverse  Magnetostrictive  Effect 

Stress  ->  Magnetization  rotation 
->  Resistivity  change,  AR/R 

Or  Piezoresistive  Effect 


COMSOL 

Simulation 


Si3N4 

Beam 


Cross-section  View 
(MEMS)  , 


Silicon 

Pillar 


>  Si3N4 


5.  Voltage  change  from  AR/R 


Pillar 

Values 

Length 

50  um 

Width 

50  um 

Height 
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Beam 

Values 

Length 

350  um 

Width 

52  um 

Thickness 

1  um 
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Bottom  View  of  Beams 


(MR)  Air  Flow  Sensor 

Ni86Fei4  Sensing  Elements 


stress 


Hk  sin(<? 


Magnetization 


,'d 


4  xIO  array  in  series  for  larger  output  signal 
Square  shape  for  avoiding  demagnetizing  field 
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MR  signal  is  related  to  beam  stress 
and  thus  air  flow  velocity. 
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Design  Comparison 


Sensing  Elements  Lsensor  Wsensor 

^sensor 

Magneto-resistance  4umx10  4um  x4 

Piezo-resistance  50um  25um 

25nm 

40nm 

Strain  1e-5 

Magneto-resistance 

Piezo-resistance 

Power 

1  mW 

1  mW 

Resistivity 

15e-8  ohm-rn  (Ni84Fe16) 

2e-5  ohm-m  (PolySi) 

Resistance 

240  ohm 

1000  hm 

Voltage  &  Current 

0.5  V,  2  mA 

1  V,  1  mA 

Current  Density 

2e10  A/m2 

1e9  A/m2 

Resistance  Change 

0.11  %  (AMR),  0.44  %  (GMR) 

0.029  % 

Voltage  Change 

0.55  mV  (AMR),  2.2  mV  (GMR) 

0.29  mV 

Johnson  Noise 

2  nV/V Hz 

4  nV/VHz 

Anisotropic  magnetoresistive  (AMR)  and  giant  magnetoresistive  (GMR)  air  flow  sensors  with  1  mW  power 
consumption  are  feasible  and  outperform  similar  piezoresistive  air  flow  sensors. 
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Coercivity 


Beak  and/or  visual  cortex  contains 
superparamagnetic  particles  to 
track/see  magnetic  flux  lines 


iviagneioeieciric  sensors  ror  ueieciing 
Magnetic  Field  (Carman’s  Group) 


http://www.usswisconsin.org/ 


Detecting  incoming  threats  using 
magnetic  perturbation 


Develop  sensitive 
magnetometer  using 
biological  inspiration  & 
phenomena  present  only 
at  nanoscale 


sensor 


Piezoelectricity 
Curie,  1&80 


Spaldin,  Fiebig, 
Science  15  July  2005 
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Normalized  Magnetization 


Method  of  Approach 


Nanoscale  Magnetoelectric  Materials  for  Detecting  Magnetic  Fields 
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200 1  -  Giant  magnetoelectric  in  bulk  composite  (Ryu) 

>  1000  papers 

2004  -  Magnetoelectric  in  thin  film 
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2007  -  Magnetoelectric  in  SD  (UCB  and  UCLA) 
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Method  of  Approach 

Magnetoelectric  Control  of  Superparamagnetism 


Ni 

nanoparticles 


(Oil)  PMN-PT 


Average  particle  size  16  nm 


TEM 


Superparamagnetic  Single  Domain 


•  Magnetoelectric  composite  induces  strain  in  Ni  nanoparticles 

•  E=0  produces  superparamagnetic  behavior 

•  E=0.4  MV/m  produces  single  domain  structure 


Magnetoelectric  composite 
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nanofiber  mesh 


strain  sensor 


Nano-Strain  Sensors  (Servati  &  Ko’s  Group) 


•  Strain  sensors  based  on  electrospun  nanofibers. 

•  Core-shell  nanofibers  for  ultra-sensitive  strain 
monitoring. 


Au  shell 


* 

PAN  core 

Core-shell  (Au-PAN)  conductive  nanofiber  strain 

mesh. 


SEM  photomicrograph  of  core-shell  (Au- 
PAN)  conductive  nanofiber  mesh. 


Cross  section  of  core-shell  (Au-PAN) 
nanofiber  mesh  in  PDMS. 
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Accomplishments:  Stable,  High-Sensitivity  Response  for 
Planar  Strain  and  Vibrational  Monitoring 
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Change  in  resistance  and  gauge  factor  K 
under  uniaxial  tensile  strain. 


Measured 
changes  in 
resistance  due  to 
vibrations  of  a 
rigid  metallic 
blade,  showing 
both  tensile  and 
compressive 
strain  sensing 
response. 


2,400- 


Stable  change  in 
resistance  over 
1 000  repeated 
stretching  and 
unloading  of  the 
sensor. 


Vibrations 
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Data  flow  and  CU  program  overview 


mg  with  sensors  and  actuators 
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Precision  signal  processing 
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Precision  interrogation  results 


Multiple  (100’s) 
sensor  precision 
ranging  supported 
by  single  network. 


Peak  #  0 


Peak  Selection  and  Measurements 
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Amplitude  Mean  Std.Dev. 


15952.2292  15941.1658  8.6323 


15239.2095  15235.7026  6.5677 


0  0,0000 


Ref.  Peak  Range  (ps)  Mean  Std,  Dev, 

None  i  3471.8773  3471.8781  0.00093601  X 

0  J  33.064  33.064  0,00000507  < 

Hone  £  0  0  0,00000000  x 

None  t  0  0  0.00000000  X 


Higher  bit-depth  DAQ 

•  New  noise  floor  =  3.0 
attoseconds 

•  Range  uncertainty  =  ± 
Angstroms  in  silicon 
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Material  Development  for  Reasoning 


AFOSR-MURI 

Bio-inspired  Sensory  Network 


Nwr 

New  York  Jhstjtute 
of  Technology 


Neuron  Circuits  for  Stretchable  Network 

(Chen’s  Group) 


In  this  project,  we  plan  to  develop  electronic  neuron  circuits  based  on  carbon 
nanotube/polymer  composites,  and  integrate  the  neuron  circuits  with  sensing 
networks  that  can  (1)  promptly  process  a  large  amount  of  signals  in  parallel  to 
recognize  exogenous  threats  accurately  and  effectively,  (2)  implement  real¬ 
time  learning  autonomously,  and  (3)  provide  dynamic  prognosis  for 
appropriate  response  for  UAV. 


Synaptic  Transistor  &  Large-scale  Neuron  Circuit 


A  synaptic  transistor  has  been 
developed  by  integrating  CNT  and 
polymer  materials  to  emulate 
biological  synapse  with  spike 
signal  processing,  learning,  and 
memory  functions. 


An  image  of  a  large  scale  neuron  circuit  by 
integrating  8192  synaptic  transistors  with  Si 
MOS  circuits  with  the  functions  of  signal  parallel 
processing,  real-time  pattern  recognition, 
adaptive  learning. 
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Dynamic  Interaction  between  Neuron  Circuit  &  System 


Neurologically  inspired  theoretical  models  and  architectures  has  been  directly 
integrated  and  applied  to  establish  the  circuit  architecture. 

The  circuits  have  been  integrated  with  the  temperature  sensing  network 
developed  at  Prof.  Chang’s  group  at  Stanford  University. 

We  will  demonstrate  (1)  promptly  process  a  large  amount  of  signals  in  parallel 
to  recognize  exogenous  threats  accurately  and  effectively,  (2)  implement 
real-time  learning  autonomously,  and  (3)  provide  dynamic  prognosis  for 
appropriate  response  for  UAV. 
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Interface  Circuits  for  PZT Actuators  (Murmann’s  Group) 

Densely  Integrated  Interface  Circuits  for  State  Sensing  Network 


□  Using  Pulse-Width-Modulation  fPWM)  to  generate  the  excitation  waveform 

□  Render  the  waveform  by  a  series  of  precisely  timed  binary  pulses 

□  High  power  efficiency:  (a)  is  bounded  by  78%;  (b)  is  bounded  by  100% 


piezo 
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Chip  layout  (to  be  taped  out  on  8/19) 

Multiphase  clock  generator 
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SRAM(PWM 
time  table) 


digital  control 
-40,000  logic 
gates 


Potential  Way 


Baseline  Generation 


Generate  large  database  of  sensor  responses  for  different  structural  states  during  training 


Record  sensor  Record  sensor  Record  sensor 

responses  at  state  ‘S1’  responses  at  state  ‘S2’  responses  at  state  ‘SN’ 


■  Enormous  amount  of  effort  &  time  consumption 

■  Next  to  impossible  to  span  entire  range  of  environmental  conditions  and 

structural  states 
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The  Proposed  Approach 


Data  Driven  a _ ^  Physics  based 

Techniques  ^  *  Strategies 


Distributed  Actuator/Sensors 


P 

Autonomous 
Guidance  units 


Decision 

Management 


Supervised  Learning 


State  Estimation 


Structural 

changes 


_  Others  .. 

Damage 

types 


Unsuoervised  learning 

(Neural-net  Architecture) 


£ 


Physics  based 
compensation  models 
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Modeling  and  Prognostics  for 
Design  and  Validation 

_ / 


Ghosh  -  Johns  Hopkins  University 
Chang  -  Stanford 
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Mechanical  and  Electromagnetic  Coupling  Modeling 


Methods  of  Approach 


Coupled  Simulation 


Multi-time  Scaling 


To  develop  an  coupled  multi-scale,  multi-physics  computational  model  and  code  for  analysis  of  electromagnetic  devices,  e.g.  sensors, 
antenna  leading  to  design 


Large  Deformation  Dynamic  Response 


Nonlinear  hyper-elastic  material 

S  =  A-tr(E^  +  2ju-E 
Finite  deformation  problem 

}  (Stl  Poi)dV  +  J  (SFT -PjdV- j  ( SuTp0b)dV 


n 


—  J*  SuT t_QdS 


Solve  for 


aa, 


U  &U 


S :  2ndPiola-Kirchhoff  Stress  Tensor 

E  :  Lagrangian  Green  Strain  Tensor 
u:  Displacement 
A,  :  Lame  Constants 
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Transient  Electromagnetic  Field 


Maxwell  equations  in  total  Lagrangian 


Scalar  and  vector  potential  in  reference 
configuration 


B  =  V  x  A  Zf  =  -V  (p- A 


1 

1 

1 

1 

1 

H(X,t)  = 

(s0J {-VO -A- (F~l  •  u) x (V x A)} •  C'1 ) x (f-1  •  u ) 
+^t{(Yx^).c} 

L  \ 

1 

1 
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Coupling  of  Mechanical  and  Electromagnetic  Field 


Methods  of  Approach 


Coupled  Simulation 


Multi-time  Scaling 


E 

5 

5- 


1 .  Electromagnetic  field  is  affected  by  the  mechanical  field 

2.  The  magnetic  potential  is  evolving  by  the  velocity  field  other  than  the  displacement  field 


Coupled  transient  electromagnetic  field  and  dynamic  field 
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Electromagnetic  field  is 
evolving  with  the 
mechanical  field 

Frequency  difference  brings  in 
significant  computational  expense 


Nrw  Yoke  Ihstjtute. 
of  technology 


Multi-physics  Spectral  Element  Method  (Chang’s  Group) 


Efficient  multi-physics  computation  tool  for  modeling  ultrasonic  waves 


Equations  of  Motion 

<r<u  +  ft  =  P«t 


Gauss’s  Law  for  Electricity 


Voltage  Input  to  Piezo  Actuator  Induced  Ultrasonic  Stress  Waves  Voltage  Output  at  Piezo  Sensor 

and  wave-crack  interaction 


Time  (ps)  Aluminum  plate  with  20  mm  cracks 


Time  (ps) 
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Design  of  Piezoelectric  Sensor  Network 


•  PESEA  :  accurate  simulations  for  a  complex  structure 

•  Genetic  algorithm:  100%  damage  detectability  with  minimum  number  of  Piezo 
actuators  and  sensors 

Integration  of  PESEA  and  GA-based  Optimization 


Given 

Structure 

>=> 

\ _ 

PESEA 

<£=> 

GA-based 

Optimization 

_ > 

■=> 

Optimal  Sensor 
Layout 
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Diagnostics  and  State  Awareness 


Chang,  Ng  -  Stanford 
Shoureshi  -  NYU 
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Sensor  Data  Processing 


— 

\ 

STAGE  3 

Prognosis,  Decision  Planning 

Autonomous  decision 

and  Control 

_ 

J 

J 

STAGE  2 

State  Quantification 


Structural  Damage 
Diagnosis 

(Type,  location  ,  extent) 


Flow  Field 
Distribution 
(temperature, 
pressure,  ^train  etc.) 


STAGE  1 

State  Classification 


o 


Motivation 


Sensor  Data  Interpretation  in  Real-time 


Sensor  Response:  St  =/(Aload,  Atemp.,  localized  damage,  ABCs,  Asensor  state  ...) 


State  field  distribution 

-Temperature 
-Pressure 

-  Air-flow 

-  Strain 

-  Structural  damage 


Corrupted 
Sensor  Data 


wvw 


How  to  accurately  assess  the  structural  state  information  from  a  network 

of  multi-functional  sensors? 
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State  Classification:  Designed  features 

Table:  Sensor  signal  measurements  under  simulated  environmental 


Number  of  Samples 

-p  ,  jponditi 

Temperature  Range 

^load  Range 

Simulated  Damage 

4  coupons 

30°C  -  95°C 

Okips  -  5kips 

0.25”  - 1.5” 

Notch  at  the  edge 

4  sensing  paths  per  actuator;  Paths  per  measurement  =16;  #  of  measurements  =1136 

Feature  Space  Representation 


O 


DS) 

O 

0> 


Peak  amplitude  change 


Peak  amplitude  change 


Difficult  to  identify  true  state  under  combined  action  of  load  and  damage; 
State  Classification  Accuracy  (logistic  regression)  <  30% 
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SL  Feature  1 


State  Classification:  Self-Learned  Features 

Unsupervised  Features  Learning:  Neural-net  based_  ‘Sparse  Auto-Encoders’ 


Feature  Space  Representation 


Wavelet  like  atoms:  1st 


Physical  Model  +  Self-Learning  Strategies 


Damage 


Temperature 


Physics-based 

Compensation 

Model 


Self-learned  features  outperform  the  self-designed  features  for  state  classification 
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Scaling  Feature  Learning 


•  Current  system  scales  better;  but  still  some 
distance  to  go  for  a  full-scale  application. 


Prior  art 


Our  previous  system 


Fly-by-feel  aircraft 


104  neurons 


1 M  parameters 
50,000  examples 


✓ 


106  neurons 


>  2M  parameters 
50,000  examples 

Local  receptive  fields, 
weight-tying 


AFOSR-MURI 

Bio-inspired  Sensory  Network 


Nwr 

New  York  Jhstjtute 
of  Technology 


■  -555^ 


Approach 


Fold  prior  MURI  work  into  extremely 
large-scale  system: 

-  Scalable  K-means  learnin 


66M  parameters 


-  Online  trainin 


5  57M  examples. 


-  Locally  connected  neurons. 

-  New  invariant-feature  learning  approach. 
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Proof-of-concept 


•  Applied  to  unlabeled  image  data. 

66M  parameters,  57M  data  points. 

(lOOOx  more  data  than  standard  benchmarks.) 


1.4  million  images. 


57  million  patches. 


_ _ 

ti=2  n  m 


Single  neuron  selects 
complex  patterns  like 


IS 


SOTtS  n  1 


3  _ 
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From  Sensing  to  Decision  Making  (Shoureshi’s  Group) 


Sensing 


Diagnostics 
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Goals 


-  To  develop  an  analytical  technique  for  observability 
and  controllability  of  large-scale,  dynamic  systems 

-To  develop  a  bio-inspired  data/information 
architecture  for  feature-based  global  diagnostics  of  a 
large-scale  system 

-  To  develop  a  bio-inspired,  feature-based  re- 
configurable  control  system  to  maintain  vehicle 
functionality  in  the  presence  of  system  failures 

-  Design  a  testbed  to  assess  MURI  team  research 
results 
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Controller  and  Diagnostics  Testbed 


Prototype  I 


COMPOSITE  PANEL 

integrated  with:  multifunctional  sensor  network 


State  Sensing: 

Stresses,  Strains,  Temp,  Pressure, 

Loads,  Damage/Failure,  Remote  Threats,  etc. 


It 

Tm  a  ■ 

¥? 
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Learning  from  others 
Learning  from  itself 


Fly-By-Feel  UA  V 


•Lower  Cost  in  Manufacturing 
•Reduced  Cost  in  Maintenance  and  Operation 
•Energy  Saving  for  Smaller  Size 
•Minimal  Human  Risk 
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Traditional  design  of  structures 
is  divided  into  a  few  disciplines 


Resulting  in 

-^overdesigned  structures 
->heavy  airplanes 

-Mime  consuming  inspections 

-^inappropriate  maintenance 


schedules 


AFOSR-MURI 

Bio-inspired  Sensory  Network 


"NYlT 

New  York  Ihstjtute, 
of  Technologic 


Technologies  developed  during  MURI 


Requires  re-thinking  the  traditional  design  strategies 
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Intelligent  design 


Concept:  “Build  confidence  on  design” 


Built-in  intelligence 


Minimize  uncertainties 
->weight  Saving 


Confidence 
on  design  and 

performance 
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Life  cycle  management 


Manufacturing 


Service 


Maintenance 
&  Repair 
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